[XPU] get_infer_param use inplace copy, remove block_tables abundant d2h copy by RuohengMa · Pull Request #7431 · PaddlePaddle/FastDeploy

RuohengMa · 2026-04-16T07:06:16Z

Motivation

get_infer_param use inplace copy, remove block_tables abundant d2h copy

Modifications

get_infer_param use inplace copy, remove block_tables abundant d2h copy

Usage or Command

None

Accuracy Tests

None

Checklist

Add at least a tag in the PR title.
- Tag list: [[FDConfig],[APIServer],[Engine], [Scheduler], [PD Disaggregation], [Executor], [Graph Optimization], [Speculative Decoding], [RL], [Models], [Quantization], [Loader], [OP], [KVCache], [DataProcessor], [BugFix], [Docs], [CI], [Optimization], [Feature], [Benchmark], [Others], [XPU], [HPU], [GCU], [DCU], [Iluvatar], [Metax]]
- You can add new tags based on the PR content, but the semantics must be clear.
Format your code, run pre-commit before commit.
Add unit tests. Please write the reason in this PR if no unit tests.
Provide accuracy results.
If the current PR is submitting to the release branch, make sure the PR has been submitted to the develop branch, then cherry-pick it to the release branch with the [Cherry-Pick] PR tag.

paddle-bot · 2026-04-16T07:06:29Z

Thanks for your contribution!

codecov-commenter · 2026-04-16T08:33:39Z

Codecov Report

❌ Patch coverage is 4.34783% with 22 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (develop@a0c39cc). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
fastdeploy/model_executor/forward_meta.py	4.34%	22 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             develop    #7431   +/-   ##
==========================================
  Coverage           ?   74.15%           
==========================================
  Files              ?      398           
  Lines              ?    55011           
  Branches           ?     8616           
==========================================
  Hits               ?    40796           
  Misses             ?    11481           
  Partials           ?     2734

Flag	Coverage Δ
GPU	`74.15% <4.34%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

PaddlePaddle-bot

🤖 AI Code Review | 2026-04-21 10:07:18

📋 Review 摘要

PR 概述：将 XPU get_infer_param 算子改为 inplace copy 模式，并将 lod_to_slot_mapping 中的 slot_mapping 计算从 CPU 移至设备端执行，消除 block_tables 的冗余 D2H 拷贝。
变更范围：custom_ops/xpu_ops/（算子实现 + 测试）、fastdeploy/model_executor/（前处理 + forward_meta）
影响面 Tag：[XPU] [OP]

📝 PR 规范检查

PR 描述过于简略，Motivation 和 Modifications 部分仅重复了标题内容，未详细描述优化方法和影响面。

描述建议（可直接复制）：

## Motivation

在 XPU 推理场景下，`get_infer_param` 每次调用都会重新分配输出 tensor 并对 block_tables 做 D2H 拷贝以计算 slot_mapping，影响推理性能。

## Modifications

1. 将 `get_infer_param` 算子由「分配新 tensor 作为输出」改为「接受预分配 tensor 进行 inplace 写入」，支持 cudagraph 模式下的 tensor 复用。
2. 将 `lod_to_slot_mapping` 中的 slot_mapping 计算逻辑从 CPU 迁移到 XPU 设备端，使用 `index_select` + `broadcast_mul` + `broadcast_add` 替代原有的 D2H + CPU 计算 + H2D 路径。
3. 更新 Python 调用侧（`xpu_pre_and_post_process.py`）和 `XPUForwardMeta`，增加 `init_inplace_tensor` 方法管理预分配 tensor 生命周期。
4. 更新所有相关测试以适配新接口。

问题

级别	文件	概述
❓ 疑问	`get_infer_param.cc:401`	`GetInferParamInferShape` 函数签名与 `PD_BUILD_OP` 声明不匹配

详情：PD_BUILD_OP 现在声明了 26 个 Inputs 和 2 个 Attrs（block_size、num_speculative_tokens），但 GetInferParamInferShape 函数仅接收 4 个 shape 参数 + 1 个 int 参数（命名为 num_speculative_tokens）。如果该函数被框架通过 PD_INFER_SHAPE 宏调用（如静态图/编译模式），存在两个风险：

参数数量不匹配可能导致未定义行为；
attrs 按序传递时，该 int 参数实际接收的是 block_size 的值而非 num_speculative_tokens，导致 slot_mapping_dec 的推断 shape 错误。

如果当前仅通过 pybind 调用（不走 PD_BUILD_OP 派发），可忽略此问题，但建议添加注释说明。

总体评价

优化思路清晰，通过 inplace tensor + 设备端计算有效减少了 D2H 拷贝和内存分配开销。测试文件同步更新，utils.py 抽取了公共初始化逻辑，代码可维护性良好。建议补充 PR 描述并确认 GetInferParamInferShape 在静态图模式下的兼容性。

RuohengMa temporarily deployed to Metax_ci April 16, 2026 07:06 — with GitHub Actions Inactive

paddle-bot bot added the XPU label Apr 16, 2026

This comment was marked as outdated.

Sign in to view

RuohengMa temporarily deployed to Metax_ci April 16, 2026 08:39 — with GitHub Actions Inactive

This comment was marked as outdated.

Sign in to view

RuohengMa had a problem deploying to Metax_ci April 16, 2026 10:45 — with GitHub Actions Failure

RuohengMa had a problem deploying to Metax_ci April 16, 2026 10:59 — with GitHub Actions Failure

This comment was marked as outdated.

Sign in to view

RuohengMa force-pushed the new_decouple_inplace_copy_block_tables branch from 8c8f7ee to 7ff2c11 Compare April 17, 2026 02:23

RuohengMa had a problem deploying to Metax_ci April 17, 2026 02:23 — with GitHub Actions Error

RuohengMa had a problem deploying to Metax_ci April 17, 2026 02:24 — with GitHub Actions Error

RuohengMa had a problem deploying to Metax_ci April 17, 2026 02:28 — with GitHub Actions Error

RuohengMa had a problem deploying to Metax_ci April 17, 2026 02:30 — with GitHub Actions Error

This comment was marked as outdated.

Sign in to view

RuohengMa had a problem deploying to Metax_ci April 17, 2026 03:09 — with GitHub Actions Failure

This comment was marked as outdated.

Sign in to view

RuohengMa had a problem deploying to Metax_ci April 17, 2026 08:00 — with GitHub Actions Error

RuohengMa had a problem deploying to Metax_ci April 17, 2026 08:18 — with GitHub Actions Failure

This comment was marked as outdated.

Sign in to view

RuohengMa force-pushed the new_decouple_inplace_copy_block_tables branch from 6d21f41 to 1dceb07 Compare April 17, 2026 09:33

RuohengMa had a problem deploying to Metax_ci April 17, 2026 09:33 — with GitHub Actions Failure

This comment was marked as outdated.

Sign in to view

RuohengMa added 4 commits April 20, 2026 02:44

inplace_copy: encoder_batch_idx/decoder_batch_idx bs == 9 ok

786c9b6

inplace_copy: encoder_seq_lod/decoder_seq_lod bs == 9 ok

e984971

inplace_copy: all bs == 9 ok

ad80381

inplace_copy: all cpu bs == 9 ok

27bd39c

RuohengMa added 19 commits April 20, 2026 02:44

inplace_copy: len_info_cpu bs == 9 ok

385a48f

finished and rm unused code

461d554

prefix_block_tables reuse

8ca7538

refine

b972890

improve performance

888e91b

remove block_table copy to cpu

a38f2ba

fix unit test

3aed097

fix

7bc1ec7

resolve conflict

0498456

refine code

7d2b8e9

fix

ddeb1b2

fix

6bc55a6

fix

3283716

fix

cbec29b

fix

fb0c429

try fix unit tests

d2430e5

fix

0d537d4

tmp save

d3b903a

fix unit test

7d3c075

RuohengMa force-pushed the new_decouple_inplace_copy_block_tables branch from 1dceb07 to 7d3c075 Compare April 20, 2026 02:55

RuohengMa had a problem deploying to Metax_ci April 20, 2026 02:55 — with GitHub Actions Failure

This comment was marked as outdated.

Sign in to view

RuohengMa added 2 commits April 20, 2026 05:53

get_infer_param try less return values

0e6ce6f

add yinwei fix

8053eda

RuohengMa had a problem deploying to Metax_ci April 21, 2026 02:03 — with GitHub Actions Failure

PaddlePaddle-bot reviewed Apr 21, 2026

View reviewed changes

Conversation

RuohengMa commented Apr 16, 2026

Motivation

Modifications

Usage or Command

Accuracy Tests

Checklist

Uh oh!

paddle-bot bot commented Apr 16, 2026

Uh oh!

This comment was marked as outdated.

Uh oh!

codecov-commenter commented Apr 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

PaddlePaddle-bot left a comment

Choose a reason for hiding this comment

📋 Review 摘要

📝 PR 规范检查

问题

总体评价

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codecov-commenter commented Apr 16, 2026 •

edited

Loading